Scene classification

ABSTRACT

An exemplary process for identifying a type of a physical environment amongst a plurality of types of physical environments is provided. The process includes obtaining, using the one or more cameras, image data corresponding to a physical environment. The process further includes identifying at least one portion of an entity in the physical environment based on the image data; determining, based on the identified at least one portion of the entity, whether the entity is an entity of a first type; determining a type of the physical environment if the entity is an entity of the first type; and presenting one or more virtual objects and a representation of the entity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional patent applicationNo. 62/657,570, entitled “MIXED REALITY CLASSIFICATION,” filed on Apr.13, 2018, the content of which is incorporated by reference for allpurposes.

BACKGROUND

The present disclosure relates generally to computer-generated realityinterfaces and, more specifically, to techniques for providingenvironment-based content using a computer-generated reality interface.

BRIEF SUMMARY

The present disclosure describes techniques for providing content usinga computer-generated reality interface depicting virtual objects incombination with a representation of a physical environment. In oneexemplary technique, image data corresponding to the physicalenvironment are obtained using one or more cameras. At least one portionof an entity in the physical environment is identified based on theimage data. Based on the identified at least one portion of the entity,whether the entity is an entity of a first type is determined. One ormore virtual objects and a representation of the entity are displayed.

DESCRIPTION OF THE FIGURES

FIGS. 1A-1B depict exemplary systems for use in variouscomputer-generated reality technologies, including virtual reality andmixed reality.

FIG. 2A depicts a user device displaying a representation of an indoorphysical environment.

FIG. 2B depicts a block diagram of a user device including classifiersconfigured to identify one or more entities of an indoor physicalenvironment.

FIGS. 2C-2G depict various flows for classifying identified entities anddetermining the type of the classified entity, according to anembodiment of the present disclosure.

FIG. 2H depicts a user device displaying a computer-generated realityinterface including virtual objects and representations of identifiedentities.

FIG. 3A depicts a user device displaying a representation of an outdoorphysical environment.

FIG. 3B depicts a block diagram of a user device including classifiersconfigured to identify one or more entities of an outdoor physicalenvironment.

FIGS. 3C-3D depict various flows for classifying identified entities anddetermining the type of the classified entity, according to anembodiment of the present disclosure.

FIG. 3E depicts a user device displaying a computer-generated realityinterface including virtual objects and representations of identifiedentities.

FIG. 4 depicts a flow chart of an exemplary technique for providingcontent in a computer-generated reality interface.

DETAILED DESCRIPTION

Various embodiments of electronic systems and techniques for using suchsystems in relation to various computer-generated reality technologiesare described.

A physical environment (or real environment) refers to a physical worldthat people can sense and/or interact with without aid of electronicsystems. Physical environments, such as a physical park, includephysical articles (or physical objects or real objects), such asphysical trees, physical buildings, and physical people. People candirectly sense and/or interact with the physical environment, such asthrough sight, touch, hearing, taste, and smell.

In contrast, a computer-generated reality (CGR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic system. In CGR, a subset of a person'sphysical motions, or representations thereof, are tracked, and, inresponse, one or more characteristics of one or more virtual objectssimulated in the CGR environment are adjusted in a manner that comportswith at least one law of physics. For example, a CGR system may detect aperson's head turning and, in response, adjust graphical content and anacoustic field presented to the person in a manner similar to how suchviews and sounds would change in a physical environment. In somesituations (e.g., for accessibility reasons), adjustments tocharacteristic(s) of virtual object(s) in a CGR environment may be madein response to representations of physical motions (e.g., vocalcommands).

A person may sense and/or interact with a CGR object using any one oftheir senses, including sight, sound, touch, taste, and smell. Forexample, a person may sense and/or interact with audio objects thatcreate a 3D or spatial audio environment that provides the perception ofpoint audio sources in 3D space. In another example, audio objects mayenable audio transparency, which selectively incorporates ambient soundsfrom the physical environment with or without computer-generated audio.In some CGR environments, a person may sense and/or interact only withaudio objects.

Examples of CGR include virtual reality and mixed reality.

A virtual reality (VR) environment (or virtual environment) refers to asimulated environment that is designed to be based entirely oncomputer-generated sensory inputs for one or more senses. A VRenvironment comprises a plurality of virtual objects with which a personmay sense and/or interact. For example, computer-generated imagery oftrees, buildings, and avatars representing people are examples ofvirtual objects. A person may sense and/or interact with virtual objectsin the VR environment through a simulation of the person's presencewithin the computer-generated environment, and/or through a simulationof a subset of the person's physical movements within thecomputer-generated environment. A virtual object is sometimes alsoreferred to as a virtual reality object or a virtual-reality object.

In contrast to a VR environment, which is designed to be based entirelyon computer-generated sensory inputs, a mixed reality (MR) environmentrefers to a simulated environment that is designed to incorporatesensory inputs from the physical environment, or a representationthereof, in addition to including computer-generated sensory inputs(e.g., virtual objects). On a virtuality continuum, a mixed realityenvironment is anywhere between, but not including, a wholly physicalenvironment at one end and virtual reality environment at the other end.

In some MR environments, computer-generated sensory inputs may respondto changes in sensory inputs from the physical environment. Also, someelectronic systems for presenting an MR environment may track locationand/or orientation with respect to the physical environment to enablevirtual objects to interact with real objects (that is, physicalarticles from the physical environment or representations thereof). Forexample, a system may account for movements so that a virtual treeappears stationary with respect to the physical ground.

Examples of mixed realities include augmented reality and augmentedvirtuality.

An augmented reality (AR) environment refers to a simulated environmentin which one or more virtual objects are superimposed over a physicalenvironment, or a representation thereof. For example, an electronicsystem for presenting an AR environment may have a transparent ortranslucent display through which a person may directly view thephysical environment. The system may be configured to present virtualobjects on the transparent or translucent display, so that a person,using the system, perceives the virtual objects superimposed over thephysical environment. Alternatively, a system may have an opaque displayand one or more imaging sensors that capture images or video of thephysical environment, which are representations of the physicalenvironment. The system composites the images or video with virtualobjects, and presents the composition on the opaque display. A person,using the system, indirectly views the physical environment by way ofthe images or video of the physical environment, and perceives thevirtual objects superimposed over the physical environment. As usedherein, a video of the physical environment shown on an opaque displayis called “pass-through video,” meaning a system uses one or more imagesensor(s) to capture images of the physical environment, and uses thoseimages in presenting the AR environment on the opaque display. Furtheralternatively, a system may have a projection system that projectsvirtual objects into the physical environment, for example, as ahologram or on a physical surface, so that a person, using the system,perceives the virtual objects superimposed over the physicalenvironment.

An augmented reality environment also refers to a simulated environmentin which a representation of a physical environment is transformed bycomputer-generated sensory information. For example, in providingpass-through video, a system may transform one or more sensor images toimpose a select perspective (e.g., viewpoint) different than theperspective captured by the imaging sensors. As another example, arepresentation of a physical environment may be transformed bygraphically modifying (e.g., enlarging) portions thereof, such that themodified portion may be representative but not photorealistic versionsof the originally captured images. As a further example, arepresentation of a physical environment may be transformed bygraphically eliminating or obfuscating portions thereof.

An augmented virtuality (AV) environment refers to a simulatedenvironment in which a virtual or computer generated environmentincorporates one or more sensory inputs from the physical environment.The sensory inputs may be representations of one or more characteristicsof the physical environment. For example, an AV park may have virtualtrees and virtual buildings, but people with faces photorealisticallyreproduced from images taken of physical people. As another example, avirtual object may adopt a shape or color of a physical article imagedby one or more imaging sensors. As a further example, a virtual objectmay adopt shadows consistent with the position of the sun in thephysical environment.

There are many different types of electronic systems that enable aperson to sense and/or interact with various CGR environments. Examplesinclude head mounted systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmounted system may have one or more speaker(s) and an integrated opaquedisplay. Alternatively, a head mounted system may be configured toaccept an external opaque display (e.g., a smartphone). The head mountedsystem may incorporate one or more imaging sensors to capture images orvideo of the physical environment, and/or one or more microphones tocapture audio of the physical environment. Rather than an opaquedisplay, a head mounted system may have a transparent or translucentdisplay. The transparent or translucent display may have a mediumthrough which light representative of images is directed to a person'seyes. The display may utilize digital light projection, OLEDs, LEDs,uLEDs, liquid crystal on silicon, laser scanning light source, or anycombination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In one embodiment, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

By virtue of displaying virtual objects in combination with arepresentation of a physical environment, electronic devices provide anintuitive CGR interface for a user to interact with his/her physicalenvironment. For example, using a CGR interface, a user can interactwith virtual objects provided in the CGR interface to perform certaintasks (e.g., control an oven or order food). One challenge forimplementing such an interface is that the virtual objects may not beprovided based on the physical environment. For example, a user may bestanding in a kitchen while virtual objects related to living roomentertainment are provided in the CGR interface. These virtual objectswould thus have limited relevance to the physical environment in whichthe user is currently located. Conventional techniques for determiningthe user's position, such as global positioning systems (GPS), typicallyhave a positioning error in the range of meters, making it difficult todetermine the precise physical environment (e.g., living room, kitchen,bedroom) of a user within, for example, a house or building.

In addition, many current techniques for determining a type of aphysical environment use entities identified in a physical environment,but are limited in that they do not distinguish amongst the types ofentities identified in the physical environment. As a result, theaccuracy of the determination of the type of the physical environmentmay be compromised or reduced. As an example, certain types of entities(e.g., a ceiling, a wall, or a table) can be found in many types ofphysical environments (e.g., kitchen, dining room, living room, etc.),and therefore are not reliable indicators of the type of the physicalenvironment. As another example, an entity that is easily movable (e.g.,a cat, a dog) is generally not a reliable indicator of the type of thephysical environment.

In accordance with some embodiments described herein, image datacorresponding to a physical environment are obtained using one or morecameras. At least one portion of an entity in the physical environmentis identified based on the image data. Based on the identified at leastone portion of the entity, whether the entity is an entity of a firsttype is determined. The type of the physical environment is thendetermined based on the entities of the first type. The first-typeentities are also referred to as inlier entities, which are generallyreliable indicators for determining the type of a physical environment.Because only entities that are generally reliable indicators for thetype of physical environment are used for determining the type of thephysical environment, the techniques described in this applicationrequire the identification of a fewer number of entities, therebyimproving the performance of identifying the type of a physicalenvironment, reducing power consumption, and enhancing operationalefficiency.

In some examples, based on the determined type of the physicalenvironment (e.g., living room, kitchen, bedroom, etc.), virtual objectsare displayed in a representation of the physical environment to provideone or more services corresponding (e.g., specific) to the type of thephysical environment. As a result, the displayed virtual objects arerelevant to the type of physical environment (e.g., living room,kitchen, bedroom) within, for example, a house or building. Accuratelyproviding services to the user in this manner enhances the userexperience and improves the performance of the system.

FIG. 1A and FIG. 1B depict exemplary system 100 for use in variouscomputer-generated reality technologies.

In some embodiments, as illustrated in FIG. 1A, system 100 includesdevice 100 a. Device 100 a includes various components, such asprocessor(s) 102, RF circuitry(ies) 104, memory(ies) 106, imagesensor(s) 108, orientation sensor(s) 110, microphone(s) 112, locationsensor(s) 116, speaker(s) 118, display(s) 120, and touch-sensitivesurface(s) 122. These components optionally communicate overcommunication bus(es) 150 of device 100 a.

In some embodiments, elements of system 100 are implemented in a basestation device (e.g., a computing device, such as a remote server,mobile device, or laptop) and other elements of the system 100 areimplemented in a head-mounted display (HMD) device designed to be wornby the user, where the HMD device is in communication with the basestation device. In some examples, device 100 a is implemented in a basestation device or a HMD device.

As illustrated in FIG. 1B, in some embodiments, system 100 includes two(or more) devices in communication, such as through a wired connectionor a wireless connection. First device 100 b (e.g., a base stationdevice) includes processor(s) 102, RF circuitry(ies) 104, andmemory(ies) 106. These components optionally communicate overcommunication bus(es) 150 of device 100 b. Second device 100 c (e.g., ahead-mounted device) includes various components, such as processor(s)102, RF circuitry(ies) 104, memory(ies) 106, image sensor(s) 108,orientation sensor(s) 110, microphone(s) 112, location sensor(s) 116,speaker(s) 118, display(s) 120, and touch-sensitive surface(s) 122.These components optionally communicate over communication bus(es) 150of device 100 c.

In some embodiments, system 100 is a mobile device. In some embodiments,system 100 is a head-mounted display (HMD) device. In some embodiments,system 100 is a wearable HUD device.

System 100 includes processor(s) 102 and memory(ies) 106. Processor(s)102 include one or more general processors, one or more graphicsprocessors, and/or one or more digital signal processors. In someembodiments, memory(ies) 106 are one or more non-transitorycomputer-readable storage mediums (e.g., flash memory, random accessmemory) that store computer-readable instructions configured to beexecuted by processor(s) 102 to perform the techniques described below.

System 100 includes RF circuitry(ies) 104. RF circuitry(ies) 104optionally include circuitry for communicating with electronic devices,networks, such as the Internet, intranets, and/or a wireless network,such as cellular networks and wireless local area networks (LANs). RFcircuitry(ies) 104 optionally includes circuitry for communicating usingnear-field communication and/or short-range communication, such asBluetooth®.

System 100 includes display(s) 120. In some examples, display(s) 120include a first display (e.g., a left eye display panel) and a seconddisplay (e.g., a right eye display panel), each display for displayingimages to a respective eye of the user. Corresponding images aresimultaneously displayed on the first display and the second display.Optionally, the corresponding images include the same virtual objectsand/or representations of the same physical objects from differentviewpoints, resulting in a parallax effect that provides a user with theillusion of depth of the objects on the displays. In some examples,display(s) 120 include a single display. Corresponding images aresimultaneously displayed on a first area and a second area of the singledisplay for each eye of the user. Optionally, the corresponding imagesinclude the same virtual objects and/or representations of the samephysical objects from different viewpoints, resulting in a parallaxeffect that provides a user with the illusion of depth of the objects onthe single display.

In some embodiments, system 100 includes touch-sensitive surface(s) 122for receiving user inputs, such as tap inputs and swipe inputs. In someexamples, display(s) 120 and touch-sensitive surface(s) 122 formtouch-sensitive display(s).

System 100 includes image sensor(s) 108. Image sensors(s) 108 optionallyinclude one or more visible light image sensor, such as charged coupleddevice (CCD) sensors, and/or complementary metal-oxide-semiconductor(CMOS) sensors operable to obtain images of physical objects from thereal environment. Image sensor(s) also optionally include one or moreinfrared (IR) sensor(s), such as a passive IR sensor or an active IRsensor, for detecting infrared light from the real environment. Forexample, an active IR sensor includes an IR emitter, such as an IR dotemitter, for emitting infrared light into the real environment. Imagesensor(s) 108 also optionally include one or more event camera(s)configured to capture movement of physical objects in the realenvironment. Image sensor(s) 108 also optionally include one or moredepth sensor(s) configured to detect the distance of physical objectsfrom system 100. In some examples, system 100 uses CCD sensors, eventcameras, and depth sensors in combination to detect the physicalenvironment around system 100. In some examples, image sensor(s) 108include a first image sensor and a second image sensor. The first imagesensor and the second image sensor are optionally configured to captureimages of physical objects in the real environment from two distinctperspectives. In some examples, system 100 uses image sensor(s) 108 toreceive user inputs, such as hand gestures. In some examples, system 100uses image sensor(s) 108 to detect the position and orientation ofsystem 100 and/or display(s) 120 in the real environment. For example,system 100 uses image sensor(s) 108 to track the position andorientation of display(s) 120 relative to one or more fixed objects inthe real environment.

In some embodiments, system 100 includes microphones(s) 112. System 100uses microphone(s) 112 to detect sound from the user and/or the realenvironment of the user. In some examples, microphone(s) 112 includes anarray of microphones (including a plurality of microphones) thatoptionally operate in tandem, such as to identify ambient noise or tolocate the source of sound in space of the real environment.

System 100 includes orientation sensor(s) 110 for detecting orientationand/or movement of system 100 and/or display(s) 120. For example, system100 uses orientation sensor(s) 110 to track changes in the positionand/or orientation of system 100 and/or display(s) 120, such as withrespect to physical objects in the real environment. Orientationsensor(s) 110 optionally include one or more gyroscopes and/or one ormore accelerometers.

FIG. 2A depicts a user device 202 displaying a representation 204 (e.g.,an image) of an indoor physical environment 200, according to variousembodiments. In the present embodiment, user device 202 is a standalonedevice, such as a hand-held mobile device (e.g., a smartphone) or astandalone head-mounted device. It should be recognized that, in otherembodiments, user device 202 can be communicatively coupled to anotherdevice, such as a base device (e.g., base device 102 b. In theseembodiments, the operations described below for providingenvironment-based content in a CGR environment can be shared betweenuser device 202 and the other device.

FIG. 2A illustrates an example in which a user 203 holds user device 202in the user's hand. In some embodiments, user 203 wears a user device asa head-mounted device. User device 202 can obtain image data using oneor more cameras. Exemplary cameras include charge-coupled device (CCD)type cameras.

As described above, in some embodiments, a CGR interface includes arepresentation of a physical environment and optionally one or morevirtual objects. In some embodiments, user device 202 presents (e.g.,displays or projects) representation 204 of indoor physical environment200 using the obtained image data. Representation 204 is a live 2D imageor 3D image of the physical environment. Representation 204 is, forexample, a representation of the physical environment from theperspective of the user device 202. In FIG. 2A, physical environment 200is at least a portion of the user's kitchen. Generally, a physicalenvironment can be an indoor environment or an outdoor environment. Inan indoor environment, a physical environment can be a specific room orarea (e.g., living room, family room, office, kitchen, classroom,cafeteria, or the like). As described in more detail below, user device202 can provide content (e.g., virtual objects) to the user based on thetype of the physical environment. For example, if the physicalenvironment is a kitchen, user device 202 can present correspondingvirtual objects, such as a food recipe, controls for operating a coffeemachine, or a user-interaction mechanism for ordering food. If thephysical environment is a living room, for example, user device 202 canpresent corresponding virtual objects, such as controls for operating aTV, a user-interaction mechanism for ordering movies, or auser-interaction mechanism for subscribing magazines.

FIG. 2B depicts a block diagram of a user device (e.g., user device 202)including classifiers 210 configured to identify one or more entities ofan indoor physical environment. As depicted in FIG. 2B, representation204 is an image captured or recorded by one or more cameras of the userdevice 202. In some embodiments, while presenting representation 204 ona display 206 of the user device, the user device performs aclassification of one or more entities of the physical environment 200using classifiers 210. Classifiers 210 can be predefined, dynamicallyupdated, and/or trained over time. In some embodiments, for an indoorphysical environment, classifiers 210 include a ceiling classifier, awall classifier, a table classifier, a chair classifier, a sinkclassifier, an animal (e.g., cat) classifier, or the like. It isappreciated that the type of classifiers used for classification ofphysical environments can be predefined based on expected use of thedevice (e.g., used at a home environment) in any desired manner.Classifiers may be used for any entity type(s) and/or granularity. Aclassifier may be used to identify a chair, or a specific type of chairs(e.g., lawn chair vs recliner) in some examples.

In some embodiments, the type of classifiers used for classification canalso be adjusted (e.g., learned or trained), for instance, using machinelearning techniques. For example, based on training data associated withdifferent physical environments, such as those in which the user devicehas been used in the past (e.g., the physical environments in which theuser device has been frequently used are living room, kitchen, etc.),the type of classifiers used for classification (e.g., ceilingclassifier, floor classifier, table classifier) can be derived ordetermined.

As illustrated in FIG. 2B, classifiers 210 identify one or more entitiesof the physical environment 200, including, but not limited to, a sink211A, a cat 211B, a wall 211C, a metal piece 211D, and a chair 211E. Insome embodiments, classifiers 210 identify an entire entity (e.g., cat211B) or a portion of an entity (e.g., a corner as a portion of wall211C). As described in more detail below, in some embodiments, if afirst portion of an entity (e.g., a leg of a chair 211E, a corner ofwall 211C, or the like) is identified and one or more properties of theentity can be determined without having to identify the entire entity,classifiers 210 can forego identifying another portion of the entity orforego identifying the entire entity. Identifying a portion of theentity, but not the entire entity, can increase identification speed,reduce power consumption, and thus improve operational efficiency of theelectronic device. In some embodiments, classifiers 210 identify one ormore entities in a physical environment based on hierarchicalclassification techniques. For example, classifiers 210 perform aninitial classification using a subset of predefined classifiers that isless than a full set of available classifiers. The initialclassification identifies one or more predefined entities. A geometriclayout of the physical environment is estimated based on the identifiedone or more predefined entities. An area is determined based on thegeometric layout and a second level classification is performed usingclassifiers corresponding to the determined area. Classifiers 210 canthus identify particular entities in the determined area. Because notall available classifiers are used for all entities, the hierarchicalclassification improves the performance of identifying particularentities in a physical environment, reduces power consumption, andenables real-time classification.

In some embodiments, based on the currently-identified at least oneportion of the entity, the user device determines whether thecurrently-identified at least one portion of the entity corresponds toat least one portion of a previously-identified entity. For example, theuser device can store data (e.g., a map) associated withpreviously-identified entities including their classifications, theirpositions, relations to each other, layouts, or the like. The userdevice can compare the data associated with previously-identifiedentities with the currently-identified at least one portion of anentity, and determine whether the currently-identified at least oneportion of the entity corresponds to that of a previously-identifiedentity. If so, the user device does not perform the determination of oneor more properties of the currently-identified entity and in turn, doesnot determine whether the currently-identified entity is of the firsttype (e.g., the entity is an inlier that can be used to determine thephysical environment the user device is facing or located in). If thecurrently-identified at least one portion of the entity does notcorrespond to at least one portion of a previously-identified entity,the user device stores data indicative of the currently-identified atleast one portion of the entity, for instance, based on which furtherentity-type and environment-type determinations are performed. As oneexample, the user device may have previously identified one or moreentities of indoor physical environment 200 as shown in representation204, such as the sink 211A. After the previous identification isperformed by the user device, cat 211B may enter the kitchen (e.g., fromanother room). Classifiers 210 of the user device can then identify cat211B and determine that cat 211B does not correspond to any previouslyidentified entity. As a result, the user device can store dataassociated with cat 211B for further determination or processing. Insome embodiments, by determining whether the currently identified entity(or a portion thereof) corresponds to a previously identified entity andperforming further determination or processing with respect to an entitythat was not previously identified, the user device reduces theconsumption of power (e.g., battery power), increases the speed ofprocessing, and thus improves the overall operational efficiency.

In some embodiments, based on the identified entity (or a portionthereof), the user device determines one or more properties of theentity. For example, based on the identified at least a portion of theentity, the user device classifies the entity and obtains properties ofthe entity from a plurality of known or learned entities. With referenceto FIG. 2C, the user device identifies a faucet and a container as aportion of sink 211A (shown in FIG. 2B). Based on the identified faucetand container, the user device classifies (e.g., using classifiers 210)the corresponding entity as a sink class entity 231A and determines oneor more properties of sink class entity 231A.

As shown in FIG. 2C, one of the entity properties determined by the userdevice is mobility, as illustrated at determination 234A. Mobility is aproperty that describes the degree to which an entity is physicallymovable (e.g., the ability to change positions over time). As an exampleshown in FIG. 2C, the user device determines that sink class entity 231Ais a fixture and thus highly stationary and/or unlikely to move.

Another entity property determined by the user device is whether theentity is a building structure or a portion thereof, as illustrated atdetermination 234B. Building structure property is an indication ofwhether an entity is a portion of a building structure (e.g., a wall, aceiling, a floor, etc.) As an example shown in FIG. 2C, the user devicedetermines that sink class entity 231A is typically not part of abuilding structure.

Another entity property determined by the user device is consistency ofan entity, as illustrated by determination 234C. Consistency indicatesthe degree to which the appearance of the entity changes over time. Asan example shown in FIG. 2C, the user device determines that theappearance of sink class 231A typically does not change over time andtherefore is highly consistent.

Another entity property determined by the user device is the likelihoodof an erroneous classification of the entity, as illustrated bydetermination 234D. As described above, based on the identified at leasta portion of the entity, the user device classifies the correspondingentity. For example, based on the identified faucet and container, theuser device classifies the entity as a sink class entity 231A. Theclassification can be associated with a likelihood of error in someexamples. For instance, a faucet and a container can also be associatedwith another entity other than a sink. Thus, in some embodiments, theuser device can estimate a likelihood of an erroneous classification(e.g., based on properties of the entity, the class of the entity,and/or other entities in the physical environment). As an example shownin FIG. 2C, the user device determines that the likelihood of anerroneous classification of sink class entity 231A is low. While FIG. 2Cillustrates four types of properties that the user device determines, itis appreciated that any number of other types of properties can also bedetermined.

With reference to FIG. 2C, in some embodiments, based on the one or moredeterminations (e.g., determinations 234A-D) of the entity properties,the user device determines whether the entity is an entity of a firsttype. An entity of the first type is, in some examples, also referred toas an inlier entity, which is an entity that can be used for determininga physical environment associated with the entity (e.g., a physicalenvironment in which the entity is located).

In some embodiments, to determine whether the entity is an entity of thefirst type, the user device determines whether a combination of the oneor more properties exceeds a confidence threshold. For example, the userdevice can determine, for each entity property, a property value or ascore. Using the example shown in FIG. 2C, the user device determinesthat the mobility property value for sink class entity 231A is low (or acorresponding numerical value), indicating that sink class entity 231Ais stationary. The user device may also determine that the buildingstructure property value for sink class entity 231A is negative (or acorresponding numerical value), indicating that sink class 231A is not abuilding structure. The user device may also determine that theconsistency property value for sink class entity 231A is high (or acorresponding numerical value), indicating that the appearance of sinkclass 231A is highly consistent over time. The user device may alsodetermine that the likelihood of an erroneous classification for thesink class entity 231A is low (or a corresponding numerical value),indicating that the confidence of a correct classification for sink 211Ais high. It is appreciated that the user device may determine propertyvalues or scores of one or more of properties and/or determine propertyvalues of any additional properties.

In some embodiments, the user device can also determine a total count ofthe properties. As described above, the user device may use one or moreproperties of the entity to determine whether the entity is of the firsttype. In the example shown in FIG. 2C, one or more properties includingmobility, building structure, consistency, and erroneous classificationand/or any other properties (not shown) can be used for determiningwhether the entity is of the first type. If all the properties are usedand no other properties are used, the user device determines that thetotal count of the properties is four.

Based on the combination of the property values or scores and the totalcount of the properties, the user device can determine whether theentity is an entity of the first type. In some embodiments, to determinethe type of the entity, the user device determines whether thecombination of the one or more property values and the total count ofthe one or more properties exceed a confidence threshold. The confidencethreshold can be configured based on a comparison of the propertyvalues/scores to a type-indicator criteria and the total count of theproperty values. For example, the confidence threshold can be satisfiedwith a relatively small count of property values if most or all of theproperty values of the one or more properties satisfy the type-indicatorcriteria. Using the example illustrated in FIG. 2C, the user devicedetermines that all property values of the properties (e.g., mobility islow, building structure is negative, consistency is high, and thelikelihood of an erroneous classification is low) satisfy thetype-indicator criteria. For example, an entity that is stationary, isnot part of a building structure, does not change its appearance overtime, and is less likely to be erroneously classified is typically agood and reliable indicator of the physical environment (e.g., kitchen)in which the entity is located. The user device therefore determinesthat the entity type (e.g., inlier or outlier) can be determined basedon these 4 (or less) properties (e.g., total count of 4 or less).

In some embodiments, to exceed the confidence threshold when at leastsome of the property values do not satisfy the type-indicator criteria,a relatively large number of property values may be required for areliable or accurate determination of the type of the entity. Theconfidence threshold for determining the type of the entity can thus beconfigured to require, for example, at least three properties with allproperties satisfying the type-indicator criteria, at least fiveproperties with some of the property values at or slightly below thetype-indicator criteria, or at least ten properties with some of theproperty values significantly below the type-indicator criteria. In someembodiments, the confidence threshold and/or criteria for determining atype of an entity can be dynamically updated or learned (e.g., throughtraining of a machine learning model).

In some embodiments, in accordance with a determination that thecombination of the one or more property values and the total count ofthe properties satisfies (e.g., exceeds) the confidence threshold fordetermining the type of the entity, the user device determines that theentity is of the first type. Continuing with the example shown in FIG.2C, if the confidence threshold is configured to be at least threeproperties with all property values above the confidence level criteria,the user device determines that the entity of sink 211A (correspondingto sink class entity 231A) is an entity of the first type (e.g.,inliers), which can be used for determining the physical environmentassociated with the entity (e.g., the physical environment of akitchen).

In some embodiments, in accordance with a determination that thecombination of the one or more property values and the count of the oneor more properties does not satisfy (e.g., does not exceed) theconfidence threshold, the user device determines that the entity is of asecond type that is different from the first type. A second-type entityis, in some examples, also referred to as an outlier, which cannot beused to determine the physical environment associated with the entity.Identification and classification of a second-type entity is describedbelow in more detail in FIGS. 2D-2F.

As described above with respect to FIG. 2B, classifiers 210 identify atleast a portion of cat 211B. Thereafter, as shown in FIG. 2D, the userdevice determines one or more properties of cat 211B based on theidentified at least a portion of cat 211B. The user device identifies,for example, a leg, a tail, and/or whiskers of the cat 211B. Based onthe identified leg, tail, and/or whiskers, the user device classifies(e.g., using classifier 210) the corresponding entity as a cat classentity 231B and determines properties of cat class entity 231B. In someembodiments, the user device determines properties without firstclassifying the corresponding entity to a particular class entity. Forexample, using the identified leg of cat 211B, and without classifyingthe corresponding entity to a cat class entity, the user device candetermine the mobility property value of the cat 211B (e.g., thecorresponding entity is likely highly mobile because anything with a legis likely mobile).

FIG. 2D depicts a flow for classifying an identified entity anddetermining the type of the classified entity, according to anembodiment of the present disclosure. With reference to FIG. 2D,properties the user device determines can include mobility, buildingstructure property, consistency, likelihood of erroneous classification,and/or any other properties. For example, as illustrated indeterminations 234A-D in FIG. 2D, the user device determines that catclass entity 231B is highly mobile; is not a building structure; issomewhat consistent over time; and has a medium likelihood of erroneousclassification (e.g., something with a leg, tail, and whiskers may alsobe other animals like a tiger), respectively. Based on the determinedproperties, the user device further determines whether a combination ofthe one or more properties exceeds a confidence threshold. Continuingwith the example shown in FIG. 2D, the user device determines that theproperty value for mobility is high (or a corresponding numericalvalue), indicating that cat class entity 231B is stationary. The userdevice may also determine that the building structure property value isnegative (or a corresponding numerical value), indicating that cat classentity 231B is not a building structure. The user device may alsodetermine that the property value for consistency is medium (or acorresponding numerical value), indicating that the appearance of catclass entity 231B may change over time (e.g., as the cat gets dirty orold over time). The user device may also determine that the propertyvalue for the likelihood of an erroneous classification is medium (or acorresponding numerical value), representing that the confidence of acorrect classification for cat 211B to cat class entity 231B is in themedium range. It is appreciated that the user device may determineproperty values of one or more of properties as illustrated atdeterminations 234A-D and/or determine property values of additionalproperties.

Based on the combination of the property values and the total count ofthe properties, the user device determines whether the entity is of thefirst type by, for example, determining whether the combinationsatisfies a confidence threshold. Continuing with the example shown inFIG. 2D, the user device determines that the mobility property valuesfor cat class entity 231B does not satisfy the type-indicator criteria(e.g., a highly mobile entity is generally not a reliable indicator ofthe type of physical environment). The user device may also determinethat the building structure property value (e.g., negative) satisfiesthe type-indicator criteria; that the consistency property value (e.g.,somewhat consistent) is at or slightly above the type-indicatorcriteria; and that the likelihood of erroneous classification (e.g.,medium) is at or slightly below the type-indicator criteria. Further,the user device determines the total count of the properties used fordetermining the type of the entity. In the example shown in FIG. 2D, theuser device may use, for example, properties including mobility,consistency, and erroneous classification for such determination becausea cat class entity 231B is clearly not a building structure and thus thebuilding structure property may be less relevant and therefore given noor less weight. The total count is thus three.

As described above, to determine the entity type, the user devicedetermines whether the combination of the one or more property valuesand the count of the one or more properties satisfies a confidencethreshold. For example, if the confidence threshold is configured to beat least three properties with all property values above the confidencelevel criteria, the user device determines that cat 211B (correspondingto cat class entity 231B) is an entity of a second type (e.g., anoutlier), which cannot be used for determining the physical environmentassociated with the entity. In the example shown in FIG. 2D, an entityof cat can be present in any physical environment and is thus a lessreliable indicator for the type of physical environment.

FIG. 2E illustrates another flow for classifying an identified entityand determining the classified entity, according to an embodiment of thepresent disclosure. As described above with respect to FIG. 2B,classifiers 210 identify at least a portion of wall 211C (e.g., a cornerof wall 211C). As shown in FIG. 2E, based on the identified at least aportion of wall 211C, the user device determines one or more propertiesof the wall 211C. For example, based on the corner of wall 211C, theuser device classifies (e.g., using classifier 210) wall 211C as a wallclass entity 231C and determines properties of wall class entity 231C.

With reference to FIG. 2E and similar to those described above,properties the user device determines can include mobility, buildingstructure property, consistency, likelihood of erroneous classification,and/or any other properties. For example, corresponding to properties asillustrated at determinations 234A-D, the user device determines thatwall class entity 231C is not mobile; is a building structure; isconsistent over time; and has a medium likelihood of erroneousclassification (e.g., mistakenly classify a shelf having a corner as awall), respectively. In some embodiments, similar to those describedabove, the user device determines whether a combination of the one ormore properties exceeds a confidence threshold. Using the example shownin FIG. 2E, the user device determines that the building structureproperty value is positive (or a corresponding numerical value),indicating that wall class entity 231B is a building structure. Asdescribed above, a building structure may not be a reliable indicatorfor the type of physical environment because it is a common entity foundin many types of physical environments. The user device may alsodetermine that the mobility property value is low, that consistencyproperty value is high, and that the likelihood of an erroneousclassification is medium (or corresponding numerical values). It isappreciated that the user device may determine property values asillustrated at determinations 234A-D and/or determine property values ofadditional properties.

Based on the combination of the property values and the total count ofthe properties, the user device can determine whether the entity is ofthe first type by, for example, determining whether the combinationsatisfies a confidence threshold. Continuing with the example shown inFIG. 2E, the user device determines that the mobility property values ofmobility (e.g., stationary) satisfies the type-indicator criteria; thatthe building structure property value of (e.g., positive) does notsatisfy the type-indicator criteria; that the consistency property value(e.g., highly consistent) satisfies the type-indicator criteria; andthat the likelihood of erroneous classification (e.g., medium) satisfiesthe type-indicator criteria. Further, the user device determines thetotal count of the properties used for determining the type of theentity. In the example shown in FIG. 2E, the user device may use, forexample, properties of building structure, consistency, and erroneousclassification for such determination. The user device may also assignmore weight to the building structure property and assign less or noweight to mobility property because a building structure is clearly notmobile and thus the mobility property of mobility is less relevant.

As described above, to determine the entity type, the user devicedetermines whether the combination of the one or more property valuesand the count of the one or more properties satisfies a confidencethreshold. For example, if the confidence threshold is configured to beat least three properties with all property values exceeding thetype-indicator criteria, the user device determines that wall 211C(corresponding to wall class entity 231C) is an entity of a second type(e.g., an outlier), which cannot be used for determining the physicalenvironment associated with the entity. In the example shown in FIG. 2E,an entity of wall can be present in any indoor physical environment andis thus a less reliable indicator for the type of physical environment.

FIG. 2F depicts another flow for classifying an identified entity anddetermining the classified entity, according to an embodiment of thepresent disclosure. As described above with respect to FIG. 2B,classifiers 210 identify metal piece 211D as a portion of arefrigerator, but may not identify the entity as a refrigerator. Asshown in FIG. 2F, based on the identified metal piece 211D, the userdevice determines one or more properties of metal piece 211D. In someembodiments, to determine the properties of the entity, the user deviceclassifies (e.g., using classifier 210) the entity as a metal piececlass entity 231D and determines properties of metal piece class entity231D.

With reference to FIG. 2F, properties determined by the user device caninclude mobility, building structure property, consistency, likelihoodof erroneous classification, and/or any other properties. For example,corresponding to properties as illustrated at determinations 234A-D, theuser device determines that metal piece class entity 231D is likely notmobile; may be a building structure; is consistent over time; and has ahigh likelihood of erroneous classification (e.g., a metal piece can beattached or a portion of various different entities such as anappliance, a building structure, a shelf, a door, a knife, a cookingpot, etc.), respectively. Similar to those described above, the userdevice determines whether a combination of the one or more propertiesand a total count of the properties exceeds a confidence threshold.Using the example shown in FIG. 2F, the user device may determine thatthe likelihood of an erroneous classification is high (or acorresponding numerical value), indicating that the confidence of acorrect classification for metal piece 211D is low because a metal piececan be a portion of any entity. The user device can also determine thatthe mobility property value is low to medium; that the buildingstructure property value is somewhere between positive and negative; andthat the consistency property value 234C is high (or correspondingnumerical values). It is appreciated that the user device may determineproperty values of one or more of properties as illustrated atdeterminations 234A-D and/or determine property values of additionalproperties.

Based on the combination of the property values and the total count ofthe properties, the user device can determine whether the entity is ofthe first type. As described above, in some embodiments, the user devicedetermines whether the combination of the one or more property valuesand the count of the one or more properties satisfies a confidencethreshold. Continue with the example shown in FIG. 2F, the user devicedetermines that the mobility property value (e.g., likely stationary) isat or slightly above the type-indicator criteria; that the buildingstructure property value of (e.g., between positive and negative) is ator slightly below the type-indicator criteria; that the consistencyproperty value (e.g., highly consistent) satisfies the type-indicatorcriteria; and that the likelihood of erroneous classification (e.g.,high) does not satisfy the type-indicator criteria. Further, the userdevice determines the total count of the properties used for determiningthe type of the entity. In the example shown in FIG. 2F, the user devicemay use, for example, all properties including the mobility, buildingstructure, consistency, and erroneous classification for suchdetermination. The user device may also assign more weight to theproperty of likelihood of erroneous classification because an initialcorrect classification can be a significant factor for the subsequentdetermination of the properties.

As described above, to determine the entity type, the user devicedetermines whether the combination of the one or more property valuesand the total count of the one or more properties satisfies a confidencethreshold. For example, if the confidence threshold is configured to beat least three properties with all property values above thetype-indicator criteria, the user device determines that metal piece211D (corresponding to metal piece class entity 231D) is an entity ofthe second type (e.g., an outlier), which cannot be used for determiningthe physical environment associated with the entity. In the exampleshown in FIG. 2F, an entity of a metal piece can be present as a part ofan entity in any physical environment and is thus not a reliableindicator for the type of physical environment.

FIG. 2G illustrates another flow for classifying an identified entityand determining the type of the classified entity, according to anembodiment of the present disclosure. As described above with respect toFIG. 2B, classifiers 210 identify chair 211E of the physical environment200. As shown in FIG. 2G, based on the identified chair 211E, the userdevice determines one or more properties of chair 211E. In someembodiments, to obtain the properties of the entity, the user deviceclassifies (e.g., using classifier 210) chair 211E as a chair classentity 231E and obtains properties of chair class entity 231E.

With reference to FIG. 2G and similar to those described above,properties the user device can determine include mobility, buildingstructure, consistency, likelihood of erroneous classification, and/orany other properties. For example, corresponding to properties asillustrated at determinations 234A-D, the user device determines thatchair class entity 231E is somewhat mobile (e.g., a chair can be fixedto the floor or moved to another place); is not a building structure; isconsistent over time; and has a low likelihood of erroneousclassification, respectively. In some embodiments, similar to thosedescribed above, the user device determines whether a combination of theone or more properties and a total count of the properties exceeds aconfidence threshold. Using the example shown in FIG. 2G, the userdevice determines that the mobility property value is medium; that thebuilding structure property value is negative; that the consistencyproperty value is high; and that the likelihood of an erroneousclassification is low (or corresponding numerical values). It isappreciated that the user device may determine property values of one ormore of properties as illustrated at determinations 234A-D and/ordetermine property values of additional properties.

Based on the combination of the property values or score and the totalcount of the properties, the user device can determine whether theentity is of the first type. As described above, in some embodiments,the user device determines whether the combination of the one or moreproperty values and the total count of the one or more propertiessatisfies a confidence threshold. Continuing with the example shown inFIG. 2G, the user device determines that the mobility property values(e.g., likely stationary) is at or slightly above the type-indicatorcriteria; that the building structure property value of (e.g., negative)satisfies type-indicator criteria; that the consistency property value(e.g., highly consistent) satisfies the type-indicator criteria; andthat the likelihood of erroneous classification (e.g., low) satisfiesthe type-indicator criteria. Further, the user device determines thetotal count of the properties used for determining the type of theentity. In the example shown in FIG. 2G, the user device may use, forexample, all four properties including mobility, building structure,consistency, and likelihood of erroneous classification for suchdetermination. The user device may also assign more weight to themobility property and/or the likelihood of erroneous classification. Forexample, if chair class entity 231E is a bar-type chair class entity andis fixed to the floor, it is a significant factor for determining thetype of the physical environment (e.g., kitchen). If chair class entity231E is a regular movable chair class entity and can be moved from oneplace to another, it is a less significant factor for determining thetype of the physical environment because a regular movable chair can belocated in any physical environment.

As described above, to determine the type of the entity, the user devicedetermines whether the combination of the one or more property valuesand the count of the one or more properties satisfies a confidencethreshold. If the confidence threshold is configured to be, for example,at least four properties with all property values above the confidencelevel criteria, the user device determines that chair 211E(corresponding to chair class entity 231D) is likely an entity of thefirst type (e.g., inliers), which can be used for determining thephysical environment associated with the entity.

In some embodiments, the determination of whether the entity is anentity of the first type is based on training data. For example, basedon machine learning techniques, the user device can be trained withtraining data for determining types of various entities. The user devicecan also learn and improve entity type determination over time.

With reference to FIG. 2H, in some embodiments, based on thedetermination that one or more entities in the physical environment areentities of the first type (e.g., inliers), the user devices determinesthe type of the physical environment associated with the entities. Asdescribed above with respect to the examples shown in FIGS. 2B-2G, theuser device determines that sink 211A and chair 211E are likelyfirst-type entities (e.g., inliers), and that cat 211B, wall 211C, metalpiece 211D are second-type entities (e.g., outliers). In someembodiments, based on the determination that the entities identifiedinclude at least one first-type entity, the user device can proceed todetermine the type of the physical environment. For example, based onthe determination that the physical environment includes sink 211A, theuser device may determine that the physical environment is a kitchen. Insome embodiments, the user device may require more than one first-typeentity for determining the type of the physical environment. Forexample, the user device may identify and/or classify one or moreadditional entities (e.g., chair 211E, oven, refrigerator, countertop,microwave, stove, etc., as shown in FIG. 2B) to determine the type ofthe physical environment. Identifying more entities of the first typecan improve the accuracy of the determination and reduce the likelihoodof error, thereby improving the operating efficiency and usersatisfaction.

In some embodiments, the user device performs the determination of thetype of the physical environment by comparing the one or more entitiesof the first type (e.g., sink 211A and chair 211E) to one or moreentities determined to be associated with the kitchen type of physicalenvironment. If a number or percentage of the entities of the first typematching with entities associated with a type of the physicalenvironment is greater than a threshold number or percentage (e.g.,90%), the user device determines that the type of the physicalenvironment is the predefined type (e.g., kitchen).

In some embodiments, more than one type of physical environment mayinclude the same particular entities. For example, with reference toFIG. 2B, the user device identifies the faucet and container as sink211A and determines that sink 211A is an entity of the first type (e.g.,because it is not mobile, consistent over the time, not a buildingstructure, and likelihood of erroneous classification is low). A sink,for example, can be a kitchen sink or a bathroom sink. As a result, theuser device may not be able to determine, based solely on sink 211A, thetype of physical environment the user device is facing or located in.

In some embodiments, the user device can be configured to determine thetype of physical environment using other information in addition to theone or more entities of the first type (e.g., sink 211A). Suchadditional information includes, for example, other entities in thephysical environment if any (e.g., the oven, microwave, countertop,etc., shown in FIG. 2B), the geometric layout of the physicalenvironment, and/or context information, for instance of the electronicdevice. As one example, the user device can use other entities in thephysical environment, in addition to sink 211A, to determine that thetype of the physical environment is a kitchen. As another example, theuser device can identify a napkin holder and determine that a napkinholder is an entity of the first type, but cannot determine the type ofenvironment based solely on the napkin holder because it is movable. Theuser device can use context information, such as the frequency thenapkin holder has been moved in the past, to determine that it is rarelymoved out of the dinner room. As a result, the user device can determinethat the type of the physical environment is a dining room. In someembodiments, based on the additional information, the user device canselect one of a plurality of types of physical environments (e.g.,kitchen, bathroom) as the type of physical environment associated withthe user device.

In some embodiments, the user device may not be able to determine thetype of the physical environment (with or without additionalinformation) or may need confirmation from the user as to the type ofthe physical environment. Thus, in some examples, after the user devicedetermines a plurality of candidate physical environments, the userdevice outputs the determined physical environments (e.g., visuallyand/or audibly outputs), and receives a selection of one of theplurality of candidate physical environments from the user.

With reference to FIG. 2H, in some embodiments, based on the determinedtype of the physical environment, the user device is configured topresent one or more virtual objects corresponding to the determined typeof the physical environment. As illustrated in FIG. 2H, in someembodiments, user device 202 presents a representation 204 of thephysical environment 200, which as described, may be determined to be akitchen. Representation 204 can be, for example, a 2D image, a video, ananimation, a 3D image, or any type of visual representation of thephysical environment or particular entities of the physical environment.For example, user device 202 presents a representation of the identifiedentities (first type and second type) in the physical environment (e.g.,an image of sink 211A, an image of cat 211B, etc.).

In some embodiments, user device 202 can be configured to, whilepresenting representation 204 of the kitchen, provide one or moreservices using one or more of the virtual objects corresponding to thephysical environment. With reference to FIG. 2H, as described above, thetype of the physical environment in this embodiment is determined to bea kitchen. As a result, user device 202 can provide, for example, avirtual object 286 (e.g., a virtual remote controller) enabling the userto control the oven (e.g., set the time for baking 2 hours); and avirtual object 288 (e.g., a user-interaction mechanism) providing recipesuggestions for dinner to the user. In some embodiments, the virtualobjects can be superimposed (e.g., overlaid) on a representation.Virtual objects can also be presented in a separate display area of userdevice 202 or another device communicatively coupled to user device 202.

In some embodiments, a user device presents one or more virtual objectswithout (e.g., prior to) determining the type of the physicalenvironment. For example, the user device may identify a TV entity inthe physical environment. By way of example, in some embodiments, theuser device determines that services can be provided based on theidentified entity regardless of the type of physical environment. Forinstance, a TV guide service or TV subscription service can be providedregardless of whether the TV entity is located in a bedroom or a livingroom. Accordingly, the user device is configured to present one or morevirtual objects based on the identified TV entity (e.g., a virtualobject enabling the user to receive an on-line movie streaming service)without having to determine the type of physical environment (e.g.,whether the physical environment is a living room or bedroom).

In some embodiments, after the user device presents one or more virtualobjects, the user device receives input representing a selection avirtual object of the one or more presented virtual objects (e.g., froma user of the user device), and performs one or more tasks in accordancewith the selected virtual object. For example, the user device mayreceive a selection of virtual object 288. Based on the selection, theuser device can further present the details of the recipe suggestionsand/or hyperlinks to websites for buying ingredients of the recipe.

In some embodiments, the determination of the type of the physicalenvironment is based on some or all of the entities located in thefield-of-view of the user device. In the above examples, the entitieslocated in the field-of-view of the user device may include a sink, acat, a chair, an oven, a refrigerator, etc. The determination of thetype of the physical environment is thus based on these entities. Insome examples, the field-of-view of the user device changes as the userdevice is positioned to face another direction. The determination of thetype of the physical environment can thus be updated or re-performedbased on the entities located in the changed field-of-view. Further, thevirtual objects presented on the user device can also be updatedcorresponding to the entities in the changed field-of-view. For example,rather than presenting virtual objects 286 and 288, the user device canpresent other virtual objects corresponding to the entities located inthe changed field-of-view (e.g., a virtual object enabling the user toreceive a movie streaming service if one of the entities located in thechanged field-of-view is a TV).

While the above examples are directed to an indoor physical environment(e.g., a kitchen), it is appreciated that techniques describes above canalso be used for an outdoor physical environment. FIGS. 3A-3E depictrepresentations of entities of an outdoor physical environment, flowsfor classifying identified entities and determining the type of theclassified entities, and a CGR interface including virtual objects.

FIG. 3A depicts a user device 202 presenting a representation 304 of anoutdoor physical environment 300. FIG. 3A illustrates an example where auser 203 holds user device 202 in the user's hand. In some embodiments,user 203 wears a user device as a head-mounted device. User device 202can obtain image data using one or more cameras. Exemplary camerasinclude charge-coupled device (CCD) type cameras and event cameras.

In some embodiments, user device 202 presents representation 304 of theoutdoor physical environment 300 using the obtained image data.Representation 304 is a live 2D image or 3D image of physicalenvironment 300 from the perspective of the user device 202. In FIG. 3A,physical environment 300 is at least a portion of a park.

FIG. 3B depicts a block diagram of a user device (e.g., user device 202)including classifiers 310 configured to identify one or more entities ofan outdoor physical environment. As depicted in FIG. 3B, representation304 is an image captured or recorded by one or more cameras of the userdevice. In some embodiments, while presenting representation 304 on adisplay 306 of the user device, the user device performs classificationusing classifiers 310. Classifiers 310 can be predefined, dynamicallyupdated, and/or trained over time. In some embodiments, for an outdoorenvironment, classifiers 310 include a tree classifier, a leafclassifier, a building classifier, a lake classifier, a lawn classifier,or the like. Thus, classifier 310 can include different classifiers fromclassifier 210, which is used for indoor environment. In someembodiments, classifiers 310 identify one or more entities in a physicalenvironment based on hierarchical classification techniques describedabove. As illustrated in FIG. 3B, classifiers 310 identify one or moreentities including, but not limited to, a tree leaf 311A, a tree trunk311B, a house 311C, etc. In some embodiments, classifiers 310 identifythe entire entity (e.g., house 311C) or a portion of an entity (e.g.,tree leaf 311A and tree trunk 311B of a tree). In some embodiments,based on the identified entity (or a portion thereof), the user devicedetermines one or more properties of the entity. For example, based onthe identified portion of the entity, the user device classifies theentity and determines properties of the entity from a plurality of knownor learned entities. FIG. 3C depict a flow for classifying an identifiedentity and determining the type of the classified entity, according toan embodiment of the present disclosure. With reference to FIG. 3C, theuser device identifies one or more leafs. Based on the identified leafs,the user device classifies (e.g., using classifier 310) the entity as aleaf class entity 331A and determines properties of leaf class entity331A.

As shown in FIG. 3C, and similar to those described above, one of theproperties that the user device determines is mobility, as illustratedat determination 234A. As an example shown in FIG. 3C, the user devicedetermines that leaf class entity 331A is typically not movable (e.g.,stationary) or only slightly movable in a short distance (e.g., leavescan be moving in wind). Another property determined by the user deviceis building structure property, as illustrated at determination 234B. Asan example shown in FIG. 3C, the user device determines that leaf classentity 331A is typically not part of a building structure. Anotherproperty determined by the user device is consistency, as illustrated atdetermination 234C. As an example shown in FIG. 3C, the user devicedetermines that the appearance of leaf class entity 331A typicallychanges over time (e.g., change based on season) and therefore isinconsistent.

Another property determined by the user device is the likelihood of anerroneous classification of the entity, as illustrated at determination234D. As an example shown in FIG. 3C, the user device determines thatthe likelihood of erroneous classification of leaf class entity 331A islow. While FIG. 3C illustrates four types of properties that the userdevice determines, it is appreciated that other types of properties canalso be determined.

With reference to FIG. 3C, in some embodiments, based on the one or moredeterminations (e.g., determinations 234A-D) of the entity properties,the user device determines whether the entity is an entity of the firsttype. As described above, the user device can determine property valuesor scores for each property and a total count of the properties. Usingthe example shown in FIG. 3C, the user device determines that theconsistency property value for leaf class entity 331A is low (or acorresponding numerical value), indicating leaf class entity 331A canchange its appearance over time (e.g., change based on the season). Theuser device can further determine that the building structure propertyvalue for leaf class entity 331A is negative; that the mobility propertyvalue is low (or a corresponding numerical value); and that thelikelihood of an erroneous classification is low (or correspondingnumerical values). It is appreciated that the user device may determineproperty values of one or more of properties as illustrated atdeterminations 234A-D and/or determine property values of additionalproperties.

Based on the combination of the property values or score and the totalcount of the properties, the user device can determine whether theentity is of the first type. As described above, in some embodiments,the user device determines whether the combination of the one or moreproperty values and the count of the one or more properties satisfies aconfidence threshold. Continue with the example shown in FIG. 3C, theuser device determines that the consistency property value (e.g.,inconsistent) does not satisfy the type-indicator criteria. Typically,an entity that changes its appearance over time is not a reliableindicator of the type of physical environment. The user device furtherdetermines that the mobility property values (e.g., stationary)satisfies the type-indicator criteria; that the building structureproperty value (e.g., negative) satisfies the type-indicator criteria;and that the likelihood of erroneous classification (e.g., low)satisfies the type-indicator criteria. Further, the user devicedetermines the total count of the properties used for determining thetype of the entity. In the example shown in FIG. 3C, the user device mayuse, for example, properties including the mobility, the buildingstructure, and the consistency for such determination because a leafclass entity 331A is clearly not a building structure and thus thebuilding structure property is less relevant, and thus given no or lessweight.

As described above, to determine the type of the entity, the user devicedetermines whether the combination of the one or more property valuesand the total count of the one or more properties satisfies a confidencethreshold. The confidence threshold can be pre-configured, dynamicallyupdated, and/or learned over time. If the confidence threshold isconfigured to be at least three properties with all property valuesabove the type-indicator criteria, the user device determines that treeleaf 311A (corresponding to leaf class entity 331A) is an entity of thesecond type (e.g., an outlier), which cannot be used for determining thephysical environment associated with the entity. In the example shown inFIG. 3C, a leaf entity can change its appearance over time and thus isunlikely a reliable indicator for the type of physical environment.

FIG. 3D illustrates another flow for classifying an identified entity(e.g., tree trunk 311B) and determining the type of tree trunk classentity 331B. As shown in FIG. 3D, the user device can determine theproperties as illustrated at determinations 234A, 234B, and 234D similarto those described above with respect to FIG. 3C. For example, the userdevice determines that trunk class entity 331B is typically not movable(e.g., stationary), is not part of a building structure, and thelikelihood of erroneous classification of trunk class entity 331B islow.

Another property that the user device can determine is consistencyproperty as illustrated at determination 234C. As an example shown inFIG. 3D and unlike that in FIG. 3C, the user device determines that theappearance of trunk class entity 331B typically does not changes overtime (e.g., does not change based on season) and therefore isconsistent.

With reference to FIG. 3D, in some embodiments, based on a combinationof the one or more properties (e.g., mobility, building structure,consistency, and/or erroneous of classification) of the entity and atotal count of the properties, the user device determines whether theentity is an entity of the first type. As described above, the userdevice can determine property values or scores for each property and atotal count of the properties. Using the example shown in FIG. 3D, theuser device determines that the mobility property value for tree trunkclass entity 331B is low; that the building structure property value isnegative; that the consistency property value is high; and that thelikelihood of an erroneous classification is low (or a correspondingnumerical value), representing that the confidence of a correctclassification for trunk class entity 311B is high. It is appreciatedthat the user device may determine property values of one or more ofproperties as illustrated at determinations 234A-D and/or determineproperty values of additional properties.

In some embodiments, the user device can also determine a total count ofthe properties. In the example shown in FIG. 3C, one or more propertiesas illustrated at determinations 234A-D and/or any other properties (notshown) can be used for determining whether the entity is of the firsttype. For example, if the mobility, the consistency, and the erroneousclassification properties are used but the building structure propertyis not used (because it is less relevant and therefore given no weight),the user device determines that the total count of the properties isthree.

Based on the combination of the property values or score and the totalcount of the properties, the user device can determine whether theentity is of the first type. As described above, in some embodiments,the user device determines whether the combination of the one or moreproperty values and the count of the one or more properties satisfies aconfidence threshold. Continuing with the example shown in FIG. 3D, theuser device determines that the mobility property values (e.g.,stationary) satisfies the type-indicator criteria; that the consistencyproperty value (e.g., highly consistent) satisfies the type-indicatorcriteria; and that the likelihood of erroneous classification (e.g.,low) satisfies the type-indicator criteria.

As described above, to determine the type of the entity, the user devicedetermines whether the combination of the one or more property valuesand the total count of the one or more properties satisfies a confidencethreshold. For example, if the confidence threshold is configured to beat least three properties with all property values above the confidencelevel criteria, the user device determines that tree trunk 311B(corresponding to tree trunk class entity 331B) is an entity of thefirst type (e.g., an inlier), which can be used for determining thephysical environment associated with the entity. In the example shown inFIG. 3D, an entity of tree trunk does not change its appearance overtime and thus can be a reliable indicator for the type of physicalenvironment.

With reference to FIG. 3E and similar to those described above, based onthe determination that one or more entities in the physical environmentare entities of the first type (e.g., inliers), the user devicedetermines the type of the physical environment the user device isfacing or positioned in. As described above, in the examples shown inFIGS. 3A-3D, the user device determines that tree leaf 311A is an entityof the second type (e.g., outlier) and that tree trunk 311B is an entityof the first type (e.g., inlier). In some embodiments, based on thedetermination that the entities identified include at least one entityof the first type, the user device can proceed to determine the type ofthe physical environment. In some embodiments, the user device mayrequire more than one entity to be entities of the first type before itcan determine the type of the physical environment. For example, theuser device may identify additional entities (e.g., house 311C, a lake,a lawn, etc.) and determine that one or more of these additionalentities are entities of the first type before it determines the type ofthe physical environment. In some embodiments, the user device can beconfigured to determine the type of physical environment using otherinformation in addition to the one or more entities of the first type(e.g., tree trunk 311B). For example, the user device can use datacollected from a GPS sensor to assistant determining the type of thephysical environment shown in FIG. 3E (e.g., the GPS sensor indicatesthat tree trunk 311B is located within an area of a park).

With reference to FIG. 3E, in some embodiments, based on the determinedtype of the physical environment, the user device is configured topresent one or more virtual objects corresponding to the determined typeof the physical environment. As illustrated in FIG. 3E, in someembodiments, user device 202 presents a representation 304 of physicalenvironment 300 (e.g., a park). Representation 304 can be, for example,a 2D image, a video, an animation, a 3D image, or any type of visualrepresentation of the physical environment or particular entities of thephysical environment. For example, user device 202 presents arepresentation of the identified entities (first type and/or secondtype) in the physical environment (e.g., a representation of tree leaf311A, a representation of tree trunk 311B, etc.).

In some embodiments, user device 202 can be configured to, whilepresenting representation 304 of the park, provide one or more servicesusing one or more virtual objects corresponding to the physicalenvironment. With reference to FIG. 3E, as described above, the type ofthe physical environment in this embodiment is determined to be a park.As a result, user device 202 can provide, for example, a virtual object386 (e.g., a user-interaction mechanism) enabling the user to orderticket of a concert in the park.

As described above, physical environments (e.g., indoor environment oroutdoor environment) may include a variety of entities. Some of theseentities are transitory items that may not be reliable indicators fordetermining the type of physical environment. Such transitory items(e.g., a cat, a vehicle) can have high mobility relative to other, morerelatively stationary items (e.g., a building, a tree). In someembodiments, transitory items are not used for determining the type ofphysical environment.

Turning now to FIG. 4, a flow chart of exemplary process 400 foridentifying a type of a physical environment amongst a plurality oftypes of physical environments. In the description below, process 400 isdescribed as being performed using a user device (e.g., device 100 a or202). The user device is, for example, a handheld mobile device or ahead-mounted device. It should be recognized that, in other embodiments,process 400 is performed using two or more electronic devices, such as auser device that is communicatively coupled to another device, such as abase device. In these embodiments, the operations of process 400 aredistributed in any manner between the user device and the other device.Further, it should be appreciated that the display of the user devicecan be transparent or opaque. Although the blocks of process 400 aredepicted in a particular order in FIG. 4, it should be appreciated thatthese blocks can be performed in any order. Further, one or more blocksof process 400 can be optional and/or additional blocks can beperformed.

At block 402, image data corresponding to a physical environment areobtained using the one or more cameras.

At block 404, at least one portion of an entity in the physicalenvironment is identified based on the image data. In some embodiments,identifying at least one portion of an entity in the physicalenvironment includes using a plurality of entity classifiers. In someembodiments, identifying at least one portion of an entity in thephysical environment includes identifying a first portion of the entitywithout identifying the entire entity and foregoing identifying a secondportion of the entity.

At block 406, based on the identified at least one portion of theentity, whether the entity is an entity of a first type is determined.In some embodiments, based on the identified at least one portion of theentity and prior to determining whether the entity is an entity of thefirst type, whether the at least one portion of the entity correspondsto at least one portion of a previously identified entity is determined.In accordance with a determination that the at least one portion of theentity does not correspond to at least one portion of apreviously-identified entity, data indicative of the at least oneportion of the entity are stored.

In some embodiments, determining, based on the identified at least oneportion of the entity, whether the entity is an entity of the first typeincludes determining, based on the identified at least one portion ofthe entity in the physical environment, one or more properties of theentity; and determining, based on the one or more properties of theentity, whether the entity is an entity of the first type. The one ormore properties of the entity can include, for example, mobility of theentity, an indication of whether the entity is a building structure,consistency of the appearance of the entity, and/or the likelihood oferroneous classification.

In some embodiments, determining, based on the identified at least oneportion of the entity, whether the entity is an entity of the first typeincludes determining whether a combination of the one or more propertiesof the entity exceeds a confidence threshold.

At block 408, in accordance with a determination that the entity is anentity of the first type, a type of the physical environment isdetermined based on the entity. In some embodiments, determining thetype of the physical environment includes determining whether the entitycorresponds to at least one of a plurality of types of the physicalenvironments. In accordance with a determination that the entitycorresponds to at least one of the plurality of types of the physicalenvironments, one of the plurality of types of the physical environmentsis selected. In some embodiments, determining the type of the physicalenvironment further includes determining one or more additional types ofthe physical environments; presenting the determined one or moreadditional types of the physical environments; and receiving, from auser, a selection of one of the determined types of the physicalenvironments.

At block 410, one or more virtual objects and a representation of theentity corresponding to the determined type of the physical environmentare presented.

In some embodiments, input representing a selection of a virtual objectof the one or more presented virtual objects is received. One or moretasks in accordance with the selected virtual object are performed.

As described above, one aspect of the present technology is thegathering and use of data available from various sources to improve theperformance of identifying the type of physical environment the user isassociated with (e.g., located in) and providing information or servicesto the user based on the identified type of physical environment. Thepresent disclosure contemplates that in some instances, this gathereddata may include personal information data that uniquely identifies orcan be used to contact or locate a specific person. Such personalinformation data can include demographic data, location-based data,telephone numbers, email addresses, twitter IDs, home addresses, data orrecords relating to a user's health or level of fitness (e.g., vitalsigns measurements, medication information, exercise information), dateof birth, or any other identifying or personal information.

The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used toproviding customized information or services to the user. Accordingly,use of such personal information data enables users to receive morecustomized and/or personalized information or services. Further, otheruses for personal information data that benefit the user are alsocontemplated by the present disclosure. For instance, health and fitnessdata may be used to provide insights into a user's general wellness, ormay be used as positive feedback to individuals using technology topursue wellness goals.

The present disclosure contemplates that the entities responsible forthe collection, analysis, disclosure, transfer, storage, or other use ofsuch personal information data will comply with well-established privacypolicies and/or privacy practices. In particular, such entities shouldimplement and consistently use privacy policies and practices that aregenerally recognized as meeting or exceeding industry or governmentalrequirements for maintaining personal information data private andsecure. Such policies should be easily accessible by users, and shouldbe updated as the collection and/or use of data changes. Personalinformation from users should be collected for legitimate and reasonableuses of the entity and not shared or sold outside of those legitimateuses. Further, such collection/sharing should occur after receiving theinformed consent of the users. Additionally, such entities shouldconsider taking any needed steps for safeguarding and securing access tosuch personal information data and ensuring that others with access tothe personal information data adhere to their privacy policies andprocedures. Further, such entities can subject themselves to evaluationby third parties to certify their adherence to widely accepted privacypolicies and practices. In addition, policies and practices should beadapted for the particular types of personal information data beingcollected and/or accessed and adapted to applicable laws and standards,including jurisdiction-specific considerations. For instance, in the US,collection of or access to certain health data may be governed byfederal and/or state laws, such as the Health Insurance Portability andAccountability Act (HIPAA); whereas health data in other countries maybe subject to other regulations and policies and should be handledaccordingly. Hence different privacy practices should be maintained fordifferent personal data types in each country.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof providing personalize or customized services, the present technologycan be configured to allow users to select to “opt in” or “opt out” ofparticipation in the collection of personal information data duringregistration for services or anytime thereafter. In another example,users can select not to provide personal information (e.g., recentlyviewed movies in a living room) for receiving services. In yet anotherexample, users can select to limit the length of time personalinformation is maintained or entirely prohibit the development of abaseline personal preferences profile. In addition to providing “opt in”and “opt out” options, the present disclosure contemplates providingnotifications relating to the access or use of personal information. Forinstance, a user may be notified upon downloading an app that theirpersonal information data will be accessed and then reminded again justbefore personal information data is accessed by the app.

Moreover, it is the intent of the present disclosure that personalinformation data should be managed and handled in a way to minimizerisks of unintentional or unauthorized access or use. Risk can beminimized by limiting the collection of data and deleting data once itis no longer needed. In addition, and when applicable, including incertain health related applications, data de-identification can be usedto protect a user's privacy. De-identification may be facilitated, whenappropriate, by removing specific identifiers (e.g., date of birth,etc.), controlling the amount or specificity of data stored (e.g.,collecting location data a city level rather than at an address level),controlling how data is stored (e.g., aggregating data across users),and/or other methods.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, information orservices can be selected and delivered to users by inferring preferencesbased on non-personal information data or a bare minimum amount ofpersonal information, such as the content being requested by the deviceassociated with a user, other non-personal information available to theuser device providing services, or publicly available information.

The foregoing descriptions of specific embodiments have been presentedfor purposes of illustration and description. They are not intended tobe exhaustive or to limit the scope of the claims to the precise formsdisclosed, and it should be understood that many modifications andvariations are possible in light of the above teaching.

What is claimed is:
 1. An electronic device, comprising: one or moreprocessors; one or more cameras; and memory storing one or more programsconfigured to be executed by the one or more processors, the one or moreprograms including instructions for: obtaining, using the one or morecameras, image data corresponding to a physical environment; based onthe image data, identifying at least one portion of a first entity froma set of one or more entities in the physical environment; determining,based on the identified at least one portion of the first entity,whether the first entity is of a first type; in accordance with adetermination that the first entity is of the first type, determining atype of the physical environment based on the first entity; presentingone or more virtual objects and a representation of the first entitycorresponding to the determined type of the physical environment,wherein a virtual object of the one or more virtual object performs oneor more tasks corresponding to the physical environment; receiving a setof one or more inputs representing a selection of the virtual object ofthe one or more presented virtual objects; and in response to receivingthe set of one or more inputs representing the selection of the virtualobject, performing the one or more tasks corresponding to the physicalenvironment, wherein performing the one or more tasks includes changinga state of a second entity from the set of one or more entities in thephysical environment.
 2. The electronic device of claim 1, the one ormore programs further comprising instructions for: prior to determining,based on the identified at least one portion of the first entity,whether the first entity is of the first type: determining whether theat least one portion of the first entity corresponds to at least oneportion of a previously-identified entity; and in accordance with adetermination that the at least one portion of the first entity does notcorrespond to at least one portion of a previously-identified entity,storing data indicative of the at least one portion of the first entity.3. The electronic device of claim 1, wherein identifying the at leastone portion of the first entity in the physical environment comprises:identifying a first portion of the first entity without identifying theentire first entity; and foregoing identifying a second portion of thefirst entity.
 4. The electronic device of claim 1, wherein determining,based on the identified at least one portion of the first entity,whether the first entity is of the first type comprises: determining,based on the identified at least one portion of the first entity in thephysical environment, one or more properties of the first entity; anddetermining, based on the one or more properties of the first entity,whether the first entity is of the first type.
 5. The electronic deviceof claim 4, wherein determining, based on the identified at least oneportion of the first entity, the one or more properties of the firstentity comprises: classifying the first entity based on the identifiedat least one portion of the first entity; and obtaining, based on theclassification of the first entity, one or more properties of the firstentity from a plurality of properties.
 6. The electronic device of claim4, wherein the one of more properties of the first entity comprise atleast one of: mobility of the first entity; an indication of whether thefirst entity is a building structure; and consistency of the appearanceof the first entity.
 7. The electronic device of claim 4, wherein thedetermining, based on the one or more properties of the first entity,whether the first entity is of the first type comprises: determiningwhether a combination of the one or more properties of the first entityexceeds a confidence threshold.
 8. The electronic device of claim 1,wherein determining the type of the physical environment based on thefirst entity comprises: determining whether the first entity correspondsto at least one of a plurality of types of the physical environments;and in accordance with a determination that the first entity correspondsto at least one of the plurality of types of the physical environments,selecting one of the plurality of types of the physical environments. 9.The electronic device of claim 8, the one or more programs furthercomprising instructions for: determining one or more additional types ofthe physical environments; presenting the determined one or moreadditional types of the physical environments; and receiving, from auser, a selection of one of the determined types of the physicalenvironments.
 10. A non-transitory computer-readable storage mediumstoring one or more programs configured to be executed by one or moreprocessors of an electronic device with one or more cameras, the one ormore programs including instructions for: obtaining, using the one ormore cameras, image data corresponding to a physical environment; basedon the image data, identifying at least one portion of a first entityfrom a set of one or more entities in the physical environment;determining, based on the identified at least one portion of the firstentity, whether the first entity is of a first type; in accordance witha determination that the first entity is of the first type, determininga type of the physical environment based on the first entity; presentingone or more virtual objects and a representation of the first entitycorresponding to the determined type of the physical environment,wherein a virtual object of the one or more virtual object performs oneor more tasks corresponding to the physical environment; receiving a setof one or more inputs representing a selection of the virtual object ofthe one or more presented virtual objects; and in response to receivingthe set of one or more inputs representing the selection of the virtualobject, performing the one or more tasks corresponding to the physicalenvironment, wherein performing the one or more tasks includes changinga state of a second entity from the set of one or more entities in thephysical environment.
 11. The computer-readable storage medium of claim10, wherein identifying the at least one portion of the first entity inthe physical environment comprises: identifying a first portion of thefirst entity without identifying the entire first entity; and foregoingidentifying a second portion of the first entity.
 12. Thecomputer-readable storage medium of claim 10, wherein determining, basedon the identified at least one portion of the first entity, whether thefirst entity is of the first type comprises: determining, based on theidentified at least one portion of the first entity in the physicalenvironment, one or more properties of the first entity; anddetermining, based on the one or more properties of the first entity,whether the first entity is of the first type.
 13. The computer-readablestorage medium of claim 12, wherein determining, based on the identifiedat least one portion of the first entity, the one or more properties ofthe first entity comprises: classifying the first entity based on theidentified at least one portion of the first entity; and obtaining,based on the classification of the first entity, one or more propertiesof the first entity from a plurality of properties.
 14. Thecomputer-readable storage medium of claim 12, wherein the one of moreproperties of the first entity comprise at least one of: mobility of thefirst entity; an indication of whether the first entity is a buildingstructure; and consistency of the appearance of the first entity. 15.The computer-readable storage medium of claim 12, wherein thedetermining, based on the one or more properties of the first entity,whether the first entity is of the first type comprises: determiningwhether a combination of the one or more properties of the first entityexceeds a confidence threshold.
 16. The computer-readable storage mediumof claim 10, wherein determining the type of the physical environmentbased on the first entity comprises: determining, whether the firstentity corresponds to at least one of a plurality of types of thephysical environments; and in accordance with a determination that thefirst entity corresponds to at least one of the plurality of types ofthe physical environments, selecting one of the plurality of types ofthe physical environments.
 17. The computer-readable storage medium ofclaim 16, the one or more programs further comprising instructions for:determining one or more additional types of the physical environments;presenting the determined one or more additional types of the physicalenvironments; and receiving, from a user, a selection of one of thedetermined types of the physical environments.
 18. A method comprising:at an electronic device with one or more processors, memory, and one ormore cameras: obtaining, using the one or more cameras, image datacorresponding to a physical environment; based on the image data,identifying at least one portion of a first entity from a set of one ormore entities in the physical environment; determining, based on theidentified at least one portion of the first entity, whether the firstentity is of a first type; in accordance with a determination that thefirst entity is of the first type, determining a type of the physicalenvironment based on the first entity; presenting one or more virtualobjects and a representation of the first entity corresponding to thedetermined type of the physical environment, wherein a virtual object ofthe one or more virtual object performs one or more tasks correspondingto the physical environment; receiving a set of one or more inputsrepresenting a selection of the virtual object of the one or morepresented virtual objects; and in response to receiving the set of oneor more inputs representing the selection of the virtual object,performing the one or more tasks corresponding to the physicalenvironment, wherein performing the one or more tasks includes changinga state of a second entity from the set of one or more entities in thephysical environment.
 19. The method of claim 18, wherein identifyingthe at least one portion of the first entity in the physical environmentcomprises: identifying a first portion of the first entity withoutidentifying the entire first entity; and foregoing identifying a secondportion of the first entity.
 20. The method of claim 18, whereindetermining, based on the identified at least one portion of the firstentity, whether the first entity is of the first type comprises:determining, based on the identified at least one portion of the firstentity in the physical environment, one or more properties of the firstentity; and determining, based on the one or more properties of thefirst entity, whether the first entity is of the first type.
 21. Themethod of claim 20, wherein determining, based on the identified atleast one portion of the first entity, the one or more properties of thefirst entity comprises: classifying the first entity based on theidentified at least one portion of the first entity; and obtaining,based on the classification of the first entity, one or more propertiesof the first entity from a plurality of properties.
 22. The method ofclaim 20, wherein the one of more properties of the first entitycomprise at least one of: mobility of the first entity; an indication ofwhether the first entity is a building structure; and consistency of theappearance of the first entity.
 23. The method of claim 20, wherein thedetermining, based on the one or more properties of the first entity,whether the first entity is of the first type comprises: determiningwhether a combination of the one or more properties of the entityexceeds a confidence threshold.
 24. The method of claim 18, whereindetermining the type of the physical environment based on the firstentity comprises: determining, whether the first entity corresponds toat least one of a plurality of types of the physical environments; andin accordance with a determination that the first entity corresponds toat least one of the plurality of types of the physical environments,selecting one of the plurality of types of the physical environments.